With the advent of digital file processing, it is possible to digitally insert objects into a video. In order to digitally insert an object into a video first an opportunity for digital object insertion must be identified. That opportunity must then be evaluated to determine the value or benefit of digitally inserting the object. If it is decided that there is sufficient benefit or value in inserting the digital object into the video, the process of digital object insertion may then begin.
Digitally inserting objects in a video may have many benefits, for example enhancing the visual effects of the video, or improving the realism of a video, or allowing more flexibility for the video after it is shot, meaning that fewer decisions need to be made regarding objects to include in scene at the stage of filming the scenes. Consequently, digital object insertion is becoming increasingly common and utilised by video makers for all manner of purposes.
However, identifying potential opportunities for digital object insert and then evaluating them is typically a time consuming and labour intensive process. Where the identification and evaluation is performed manually by a human, it can be very time consuming and human resource intensive. Where the identification and evaluation is performed by software, the computational overheads can be significant and require substantial computing resources. Some videos may have very few, if any, opportunities for sufficiently beneficial or valuable digital object insertion, but this can only be determined after carrying out the time consuming process of identification and evaluation. As digital object insertion becomes increasingly common, existing analysis and evaluation technique have an increasing detrimental effect on resource efficiency, particularly for videos of significant length, such as films or episodes/shows of television programs.
In order to explain the stages of analysis and evaluation, it is first helpful to define some terminology that may help with understanding the process. The video may comprise a series of ‘shots’. The shots may be delineated by cuts, where the camera stops recording, or where the video material is edited to give this impression. In a first step, opportunities for digital object insertion may be identified. This is often referred to as a pre-analysis pass and may be best done by identifying scenes within the video, particularly scenes shot from the same camera position. It has been known for many years how to segment a video into scenes automatically, using shot change detection. This pre-analysis may include many other processes, and may result in a sorting process where all similar shots, or shots of the same locale, are presented together. Similar shots, or shots from the same locale, where insertion opportunities are relevant and presented together are sometimes referred to as an ‘Embed Sequence’. Humans are typically good at identifying ‘good’ opportunities for digital object insertion by manual analysis. For example a jar of instant coffee may suit a kitchen scene, but the coffee jar would look out of place in a bathroom scene, or in an outdoor desert scene. By way of example, it may be decided that a kitchen worktop in a scene is good for the digital insertion of grocery products. However, analysing videos in this way, particularly long videos, may be time consuming and if few, or no, object insertion opportunities are identified, that time may be a waste of resources.
In order then to evaluate identified opportunities, it may be important to note how long the camera spends looking at the identified location for digital object insertion, for example, the kitchen worktop. For example, if it is only a fleeting shot, it is not likely that the scene represents a good opportunity for digital object insertion. On the other hand, if the scene in the kitchen is long, and the area that is suitable for object insertion is in view for this duration, it is likely that there may be significant benefit to digitally inserting an object there. Similarly, as part of the evaluation, it may also be important to determine how many times that scene is in the video. For obvious reasons, it is important to keep a temporal consistency, of having the same item in the same position every time that scene is occurs in the video.
It may be desirable to create mock ups of the digital object opportunities by rendering preview imagery (often at a lower than final resolution) which has a blue box or cylinder in the imagery to represent the (as yet unspecified) objection to be placed. This may assist in further assessing the value of the opportunity for digital object insertion, by allowing the positioning of the digitally inserted object to be visualised. It may also be desirable to create an evaluation report on the potential opportunity, listing how much total time over how many scenes the digitally inserted object can be seen. It is important to realise that the video may be part of a series of videos (for example, the video may be one episode, or show, in a series, or season, of a television programs), meaning that the same scenes, locales, and characters may reappear in each episode or show. In such a situation, the evaluation may span some or all of the episodes/shows in the series/season, as the true value of digital object insertion may be best appreciated across the multiple episodes/shows.
There may be many reasons for digitally inserting objects into videos, and many contexts in which it may be desired. In some instances, it may be desirable in order to enhance visual effects. In other instances, it may be desirable in order to include additional items in the video that were not contemplated at the time of recording the video. In other instances, particular products may be inserted in order to function as a form of advertising. Regardless of context or purpose, there may be value and/or a number of benefits in digitally inserting objects into videos, but the process of identification and evaluation is technically complex and demanding and can be an inefficient use of resources if an insufficient number of valuable/beneficial digital object insertion opportunities are identified.